In-store product detection system

ABSTRACT

A method and system create, on a portable electronic device, a real-time graphical view of an in-store product display that pinpoints the location of one or more target products on the product display. The graphical view is generated by capturing merchandise display data, such as a video feed of the in-store product display, and processing the data to detect the one or more target products based on predefined product recognition criteria uniquely associated with the respective target products.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/399,900, filed Apr. 30, 2019, which is a continuation from U.S.application Ser. No. 13/693,126, filed on Dec. 4, 2012, now U.S. Pat.No. 10,275,825, issued on Apr. 30, 2019; the disclosure of which isincorporated by reference herein.

TECHNICAL FIELD

Example embodiments of the present application generally relate to dataprocessing techniques. More specifically, the disclosure describes atechnique for automated detection of one or more target products in abrick-and-mortar store where a customer is physically present.

BACKGROUND

Shoppers sometimes find it difficult to find particular products in aphysical brick-and-mortar sales outlets, such as retail stores. It is afeature of modern-day stores, particularly retail stores such assupermarkets and hypermarkets, that a vast variety of products andproduct variants are offered, typically provided by a multitude ofdifferent brands and in different quantities/sizes.

In stores where products are provided for customer-collection onin-store displays (e.g., on display shelving that flank shoppingaisles), shoppers in general, and sporadic shoppers in particular,struggle to identify particular products on their shopping lists amongthe many available products that are simultaneously visible.

In this disclosure, “store” means a real-world establishment wheremerchandise is sold, usually at a retail basis. “In-store” refers tosomething or someone being physically present at customer/merchandiseinterface where merchandise products are displayed or presented forcollection by shoppers, regardless of whether or not the product displayis indoors. An “in-store product display” may thus include an outdoorstall (e.g., at a mall or a fleamarket), and may also include outdoorportions of the relevant store (e.g., a gardening section of asupermarket).

BRIEF DESCRIPTION OF THE DRAWINGS

Various features of the disclosure are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings in which:

FIG. 1 is a schematic view illustrating an example embodiment of aportable electronic device configured to provide automated in-storeproduct detection.

FIG. 2 is a schematic block diagram of an automated in-store productdetection system, according to an example embodiment.

FIG. 3 is a schematic block diagram of one or more product detectionapplication(s), in accordance with an example embodiment.

FIG. 4 is a schematic network diagram depicting a network system,according to one embodiment, having a client-server architectureconfigured for exchanging data over a network.

FIG. 5 is a schematic block diagram of a mobile device configured toprovide automated in-store product detection, according to anotherexample embodiment.

FIG. 6 is a flow diagram of an example method for automated in-storeproduct detection.

FIG. 7 shows a diagrammatic representation of machine in the exampleform of a computer system within which a set of instructions may beexecuted to cause the machine to perform any one or more of themethodologies discussed herein.

DETAILED DESCRIPTION

An example embodiment of the present disclosure describes dataprocessing techniques that comprises automatically recognizing a productof interest in an image captured by a mobile electronic device in astore, and displaying on a screen of a mobile electronic device anaugmented view of the image in which the product of interest ishighlighted or pinpointed.

FIG. 1 is a schematic view of an example in-store product detectionsystem 100 comprising a portable electronic device in the example formof a mobile phone 102 having hardware and software components thatenable it to facilitate in-store discovery of one or more targetproducts. FIG. 1 also shows an example in-store product display 104 of aretail outlet in the example form of a hypermarket retail store 108.

In-store product display 104 comprises product shelving on whichmultiple products 112 are displayed for selection by shoppers, e.g., bymanual removal and placement of desired products 112 in a shoppingtrolley or shopping basket. Note that the term “product” herein means aparticular type, brand, and/or size of merchandise, while individualinstances of a product may be referred to as “product items.” Aplurality of identical product items clustered together in the productdisplay 104 may therefore be described as comprising a single product112.

A customer may employ the mobile phone 102 to automatically detect oneor more target products 116. In one example embodiment, the mobile phone102 may be used to provide, on a screen 120 of the phone 102, agraphical view in the example form of an augmented reality view 124 inwhich video content captured by a camera 215 (see FIG. 2) integrated inthe phone 102 is augmented to pinpoint the one or more target products116. The augmented reality view 124 may, for example, include a locationindicator 128 with an on-screen location that coincides with respectiveon-screen limitations of the one or more target products 116. In theexample embodiment shown in FIG. 1, the location indicator 128 comprisesa user interface element in the form of a highlight box around eachidentified target product 116. In some embodiments, a single locationindicator (e.g., a single highlight box) may indicate multiple productitems of the target product 112, for example encompassing the multipleproduct items. In other embodiments, a separate location indicator maybe provided coincident with each detected instance or item of the targetproduct 116. In the example embodiment illustrated in FIG. 1, however, asingle location indicator 128 is displayed around only one of aplurality of detected items of the target product 116.

The functionalities provided by the augmented reality view 124 enables auser to walk alongside the in-store product display 104, scanning thedisplay 104 with the phone's camera 215, visually monitoring the phonescreen 120 for indication of automated detection of the one or moretarget products 116 by on-screen presentation of the location indicator128. In some embodiments, the phone 102 may generate an audible alertwhen a target product 116 is detected.

In other embodiments, the phone 102 may capture and display stillimages, e.g. digital photographs, instead of or in addition to augmentedreality video content.

Automated detection or recognition of the one or more target products116 may comprise processing image information captured by the camera 215of phone 102, to recognize or identify a visual indicator uniquelyidentifying each of the one or more target products 116. Such a visualindicator may, for example, comprise a visual product code, such as abarcode 132, carried by product packaging. In such case, video contentor other image information captured by the camera 215 may be processedto identify predefined barcodes 132 unique to the target products 116.

Instead, or in addition, the visual indicator(s) may comprise visualattributes of the product packaging 136, such as trade dress elements140 unique to respective products 112. In such instances, digital imageinformation captured by the phone 102 may be processed to identify, forexample, a brand image, product image, brand name, product name, generalpackaging design layout, or the like, printed on the product packaging136 and uniquely associated with the particular product 112.

In other embodiments, automatically detecting the one or more targetproducts 116 may comprise capturing wireless identifier signals producedby the respective products 112, e.g. comprising a radio frequencyidentification (RFID) signals produced by respective RFID tags of theproduct 112. In such case, the phone 102 may include an RFID reader thatmay scan the in-store product display 104 at a closer distance than isthe case when visual scanning is performed by use of the camera 215 ofphone 102.

Recognition criteria on which augmented product detection is based(e.g., the above-described barcodes 132 or trade dress elements 140) maybe provided to the phone 102 before shopping is commenced. The customermay, for example, provide textual input to identify the one or moretarget products 116. Based on the textual input, the product recognitioncriteria may be obtained from a product database associated with thestore 108 (see FIG. 2). In one example, product recognition criteria maybe provided over a publicly accessible network, such as the Internet,for example from a store website. Instead, or in addition, the phone 102may communicate with a store computer system (see FIG. 2) to retrievefor access the necessary product recognition criteria for the one ormore target products 116.

Instead of identifying the target products 116 by textual input (or inaddition thereto), the customer may specify target products 116 throughthe use of the phone's camera 215, for example by scanning one or morebarcodes 132 of the target products 116, and/or generating a searchquery that includes a search image in the form of a digital photographof an item of each target product 116. For example, before going to thestore 108, the customer may consult a booklet, brochure, website, orother literature that lists products available at the store 108 togetherwith their respective barcodes 132, and may scan the barcodes 132 of thedesired target products 116. Thereafter, the store 108, the targetproducts 116 thus identified may automatically be detected by the phone102 when the user scans the in-store product display 104 with the camera215 of phone 102.

In some embodiments, processing of merchandise display data captured bythe phone 102 (e.g., image information captured by the camera 215, orwireless signal information) may be processed at a separate processorwith which the phone 102 is in communication, with results of theprocessing being communicated to the phone 102 for display to theshopper.

Features of Various Example Embodiments

As exemplified by the above-discussed example system and method, variousembodiments may be realized. These include a method that comprisesreceiving merchandise display data captured by a portable electronicdevice at a store, the merchandise display data indicating physicalpresence of multiple products on a product display at the store; in anautomated operation using one or more processors, detecting a targetproduct among the multiple products on the in-store product displaybased on the merchandise display data; displaying a graphical view of atleast a part of the in-store product display on a screen of the portableelectronic device; and providing as a part of the graphical view alocation indicator that indicates an on-screen position of the targetproduct.

To this end, an example product detection system comprises a receivermodule to receive the merchandise display data, a product locator moduleto detect the target product, a display module to display the graphicalview, and an indicator module to provide, in the graphical view, thelocation indicator coincident with the target product.

Display of the location indicator is not merely to communicate to a userof the portable electronic device that the target product is present ina targeted zone of the in-store product display, but it indicates theposition of the target product on the in-store product display relativeto the physical geometry of the in-store product display from theperspective of the portable electronic device, relative to otherproducts visible in the graphical depiction, and/or relative to a fieldof view of a camera of the portable electronic device, such that nofurther visual searching for the product by a user of the portableelectronic device is needed to pinpoint the target product among themultiple products on the in-store product display.

The merchandise display data may be captured by a customer carrying theportable electronic device (e.g., a mobile phone, a tablet PC, orwearable electronics such as smart glasses with camera and displaycapabilities, or the like) while the customer is at a store in which theproduct display is located.

In some embodiments, the receiver module comprises a data capturingarrangement provided by the portable electronic device to capture themerchandise display data originating from the in-store product display,for example by capturing one or more images of the product display, orby receiving RFID signals from respective products on the productdisplay. The receiver module may be configured to capture themerchandise display data from within viewing proximity of the in-storeproduct display, e.g., being at a distance from the product display suchthat different products on the product display are visuallydistinguishable.

Processing of the merchandise display data to locate the target productmay be performed at the portable electronic device. Instead, or inaddition, the merchandising display data may be communicated via adistributed network such as the Internet to a remote or off-siteprocessor (e.g., an online server) that provides target productidentification and/or location services to a multitude of portableelectronic devices.

Detecting of the target product may thus in some embodiments compriseprocessing the merchandise display data onboard the portable electronicdevice to identify one or more product recognition criteria for thetarget product in the merchandise display data. In other embodiments,the system may be provided off-site, remote from the portable electronicdevice. In such embodiments, the receiver module may be configured toreceive merchandise display data (e.g., captured image data or capturedsignal data) from the portable electronic device, the display module andthe indicator module being configured to generate display data based onresults of processing of the merchandise display data by the productlocator module, and to transmit the display data to the portableelectronic device to enable display of the graphical view together withthe location indicator on the screen of the portable electronic device.

As previously described, some embodiments comprise capturing imageinformation by use of an onboard camera of the portable electronicdevice, the image information representing an image of at least part ofthe in-store product display, to provide the merchandise display data.In such cases, the detecting of the target product may compriseprocessing the image information to recognize a visual indicator that isunique to the target product.

The visual indicator may comprise one or more trade dress elements of apackaging of the target product and/or a visual product code carried onthe target product.

The method may comprise capturing video content at the portableelectronic device, in which case providing the graphical view maycomprise displaying a video image based on the captured video content,the video image showing at least part of the in-store product displaysubstantially in real time together with the location indicator, tocontinuously indicate the on-screen position of the target product inthe video image.

In some embodiments, capturing the merchandise display data may comprisereceiving respective wireless identifier signals from the multipleproducts, the identifying of the product recognition criteria comprisingidentifying a particular one of the identifier signals as being a uniqueidentifier signal of the target product.

One or more product recognition criteria that define one or more uniqueproduct attributes of the target product may be received at the portableelectronic device, detection of the target product being based on theproduct recognition criteria. The unique attribute may comprise aproduct code that uniquely identifies the target product. The productcode may be contained in the merchandise display data, for example as ascanned visual barcode, or as a code carried by a wireless signal, suchas an RFID tag code. Instead, or in addition, the product recognitioncriteria may comprise definition of one or more visual characteristicsor visual indicators, such as trade dress elements, that are uniquelyassociated with the target product.

The merchandise display data may include store mapping identifiers ortags that indicate an in-store position, and/or that may be uniquelymapped to a particular in-store location. RFID signals produced by or inassociation with respective products may, for example, include alocation code that is uniquely associated with the particular display,for example being mapped to the particular display in mappinginformation under central administration by a store computer system.Instead, or in addition, visual location tags may be placed in theproduct display, in which case the particular location at which adigital photograph of a product display was taken can be established byprocessing the resultant image data to identify the unique visuallocation tag. In an example, the visual location tags may be color-codedand may be largely sized (relative, e.g., to product barcodes or visualproduct codes), to facilitate ready and accurate optical recognition ofthe location tags in digital photographs taken at a distance from theproduct display broadly similar to that at which a browsing shopperwould typically view the product display.

A search query for the target product may be submitted by the customer(e.g., being transmitted to a store computer system), the productrecognition criteria being received by the portable electronic devicefrom the store computer system responsive to the search query.

In instances where a consumer has insufficient information on which tobase automatic recognition of the target product (e.g., whererecognition of the target product is to be performed with reference to aproduct code to which the consumer does not have immediate access) asearch query may be submitted to the store computer system forprocessing of the search query to identify the target product in aproduct database forming part of the store computer system. One or moreunique attributes of the target product may then be retrieved from theproduct database, and may be transmitted to the portable electronicdevice as the product recognition criteria. The search query maycomprise textual search content and/or image content.

The consumer may thus, for example, take a picture of a sample productitem of the target product or its packaging (e.g., a previouslypurchased product item) and may submit the search picture to the storecomputer system. In such case, the store computer system may perform animage-based search to identify the target product, to retrieve from itsproduct database the one or more unique product attributes that mayserve as the product recognition criteria. Such product recognitioncriteria lookup may not only facilitate automated in situ productrecognition based on store-specific product codes (in which case theconsumer need not concern herself with store-specific productinformation), but may also serve to promote ambulatory product searchingbased on up-to-date information.

For example, in instances where the unique product attribute is a visualindicator such as trade dress elements of the target product'spackaging, consumer access (via the portable electronic device) to aproduct database of the store computer system may enable the consumer toperform automated searching for the target product based on a photographof a sample target product item having outdated packaging, or that is ina size or volume that is not in stock in the store. Some embodiments ofthe system may thus provide for generation of search queries thatcomprise a search image, for example by use of the portable electronicdevice's camera.

The search image may thus in some instances define the one or moreunique product attributes, so that, for example, a picture of a sampleitem of the target product taken with the portable electronic device'scamera may serve as sufficient product recognition criteria. In otherembodiments, the search image may be submitted to the store computersystem for retrieval of product recognition criteria from the store'sproduct database.

The search query may comprise a visual product code unique to the targetproduct. In such cases, creation of the search query may be by scanningan instance of the visual product code by use of the onboard camera ofthe portable electronic device.

Multiple target products may simultaneously be the subject of automaticdetection. In such cases, the one or more product recognition criteriafor a shopping list comprise a plurality of target products. When aplurality of target products are in such instances simultaneouslydetected in a target zone captured by the camera, a respective locationindicator for each of the plurality of target products that areidentified may be provided in the graphical view.

Some embodiments may provide for automated cooperation between a homepantry system and the portable electronic device, to automaticallycreate a shopping list and download the shopping list to the portableelectronic device. The method may thus comprise receiving the one ormore product recognition criteria for the shopping list from a homepantry system of a user of the portable electronic device.

Some example embodiments may therefore comprise monitoring stock levelsof products in the home pantry system, determining that a stock levelfor a particular product in the home pantry system is below a thresholdlevel, and automatically adding the particular product to the shoppinglist. The shopping list may be communicated from the home pantry systemto the portable electronic device by transmission of the one or moreproduct recognition criteria for the respective target products formingpart of the shopping list. The home pantry system may be a computerizedsystem that is configured to monitor and manage inventory or stocklevels in the customer's home, and automatically to identify when stockof respective products is running low or is depleted and needs to bereplenished. To this end, a processor of the home pantry system may beoperatively coupled with one or more physical product storage devices,such as kitchen cupboards, a pantry, or a fridge, to automaticallyidentify products that are to be added to the shopping list, andautomatically to communicate the shopping list to the portableelectronic device.

The home pantry system may, in some embodiments, retrieve productlocation information about their respective target products on theshopping list, the product location information indicating in-storelocations of the respective target products, e.g. relative to a storemap. The product location information may be employed together withstore map information to make the location information of the respectiveproducts available to the consumer, e.g. on a screen of the portableelectronic device.

Some embodiments may include automatic plotting of an in-store route, tomap out a suggested route that passes the respective locations of eachof the shopping list target products. Such automatic mappingfunctionality may facilitate efficient in-store movement of the consumerin order to gather all of the target product on the shopping list.

In some embodiments, the method is implemented by a portable electronicdevice that comprises a product detection module to generate anaugmented reality display of multiple consumer products based on imagedata captured by a camera of the device, to detect a product of interestamong the multiple consumer products, and to dynamically pinpointproduct of interest in the augmented reality display. By “augmentedreality display” is meant display of real-time video content captured bythe portable electronic device's camera, the video content beingaugmented by one or more screen elements, such as the locationindicator(s). While some embodiments of such an augmented realitydisplay may be sensitive to the field of view of the user, so that thescreen display is more or less seamlessly integrated with thesurrounding background (thus effectively providing an “overlay” to thecustomer's perspective of the store environment in reality), the term“augmented reality” includes display of augmented real-time videocontent wherever the portable electronic device is pointed, regardlessof the user's perspective. The portable electronic device may include,in other embodiments, a wearable display screen and integrated camera(e.g., a mobile phone device integrated in a pair of glasses), so thatthe graphical view comprises a heads-up display in which the one or morelocation indicators are overlaid on the user's view through transparentlenses of the wearable display screen.

It is a benefit of the example systems and methods that it facilitatesin-store discovery of desired products by shoppers.

Example System

More specific examples of system elements and components by which thedisclosed methods may be implemented will now be described. Note thatthe systems and system components described below serve to providevarious functionalities mentioned both in the following description andelsewhere in this disclosure, and that, for clarity of description, thefunctionalities of the system components are not in all instancesexplicitly reiterated in the following description of the systemcomponents and elements.

For ease of reference, different environments of a particular systemelement may be indicated by identical reference numerals in differentfigures, so that an element indicated by the same reference numeral isnot necessarily identical in all respects.

FIG. 2 is a block diagram depicting an example embodiment of a system200 to facilitate in-store discovery of target products. The examplesystem 200 includes a portable electronic device in the example form ofa mobile phone 102 that may be carried on the person of a customer orshopper 203 at a retail store 108. The mobile phone 102 may include aprocessor 206 coupled to one or more memories 209 on which one or moremobile phone applications may be stored for execution by the processor206. The mobile phone applications may include one or more productdetection applications 212 that comprise instructions that enable themobile phone 102 to perform the respective processes and/or operationsdisclosed herein, when the instructions are executed by the processor206. The mobile phone 102 further comprises a camera 215 to captureimage information, e.g., digital video and/or digital photographs. Themobile phone 102 also has a display screen 120.

FIG. 3 is a block diagram schematically showing various example modulesprovided by the product detection application(s) 212 for implementingthe disclosed method operations. A receiver module 307 may be providedto receive merchandise display data captured by the mobile phone 102,e.g. comprising video content of an in-store display 104 on which atarget product 116 is situated. A product locator module 314 may beconfigured to detect one or more target products 116 among multipleproducts 112 that are present in a target zone of the in-store productdisplay 104 and that are thus represented in the captured merchandisedisplay data.

The product detection application(s) 212 further comprises a displaymodule 321 to display a graphical view of the in-store product display104 on the screen 120, and an indicator module 328 to provide a locationindicator 128 (see FIG. 1) in the graphical view 124, thus indicating anon-screen position of the target product 116.

Returning now to FIG. 2, the store 108 may include a store computersystem 218 with which the mobile phone 102 can communicate (eitherdirectly or indirectly) to submit search queries and/or to receiveproduct-related data (e.g., product recognition criteria) responsive tosuch search queries. The store computer system 218 may therefore includea product database 221 holding information regarding the multipleproducts 112 that are in stock in the store 108. The product database221 may include, for each product 112, product recognition criteria suchas a visual product code, visual indicators on the product's packaging(e.g., a unique image on a cereal box, a unique rendering of a tradename, or the like). The store computer system 218 may include acommunications module 224 for establishing a communication session withthe mobile phone 102, e.g. by creating an in-store local area network(LAN) to which the mobile phone 102 is connectable.

The store computer system 218 has one or more processors 228, and mayinclude a search query processing module 232 to process search queriessubmitted to the store computer system 218 by the mobile phone 102, e.g.to identify and provide product recognition criteria to the mobile phone102 responsive to a search query, as described elsewhere herein.

The store computer system 218 may further comprise one or more productdetection application(s) 212 (see FIG. 3) that may enable the processingof merchandise display data captured by the mobile phone 102 (e.g.,digital video content) for detecting one or more target products 116.The product detection applications 212 of the store computer system 218may be used for data processing remotely from customer portableelectronic devices in instances where the electronic devices (e.g., themobile phone 102) do not have sufficient processing capacity to parse,e.g., video information captured by the phone 102 substantially inreal-time.

The store computer system 218 may further comprise a store map module236 in which mapping information (e.g., a mini-map) about the layout ofthe retail store 108 and the location of respective products 112 arestored. This enables the store computer system 218, in some embodiments,to provide location information about the one or more target products116 to the mobile phone 102, in which case the automated productdetection by the mobile phone 102 may be performed based at least inpart on location information that identifies one or more in-storepositions or locations for the target product 116. The store computersystem 218 may in such cases communicate with an in-store locatingsystem 240 to monitor the position of the customer's mobile phone, thusenabling the determination by the store computer system 218 and/or themobile phone 102 of the physical location of the mobile phone 102relative to the target product 116 in the store 108. Note that althoughthe store computer system 218 is illustrated in the example embodimentof FIG. 2 as having functionalities to process video data for opticalproduct recognition, as well as the in-store locating system 240, thestore computer system 218 may, in other embodiments, omit one or more ofthese functionalities.

The store computer system 218 may be connected to a customer-accessiblenetwork 244, such as the Internet, and may be configured to providecustomer services to the shopper 203, for example by communication overthe network 244 with the store computer system 218 via the shopper'smobile phone 102, home computer system 248, or the like. The shopper 203may retrieve product information, such as product barcodes 132,distinctive visual attributes of products, or the like, from the productdatabase 221 before leaving home. In one example, the shopper 203 maycreate a shopping list comprising multiple target products 116 bybrowsing a web site provided by the store computer system 218,downloading product recognition criteria for each target product 116,and transferring the product recognition criteria to the mobile phone102 for use in automated product detection at the store 108. In otherembodiments, the store computer system 218 may be replaced with oraugmented by a merchant computer system and/or website that provides thedescribed functionalities to multiple stores 108 at different locations.One example embodiment of such a platform architecture is laterdescribed with reference to FIG. 4.

The system 200 may yet further include a home pantry system 252 toautomatically monitor stock levels or product inventory at a residenceof the shopper 203, and to automatically maintain a shopping list byadding at least some products 112 to the shopping list responsive todetecting that stock levels of those products 112 in the pantry ofshopper 203 fall below a threshold level. The home pantry system 252 maythus include one or more smart or computer-enabled appliances and/orproduct storage devices having the functionality to monitor or keeptrack of its contents.

The home pantry system 252 may be configured automatically tocommunicate with the store computer system 218 or its equivalent(s), toobtain product recognition criteria (e.g., barcodes, optical productrecognition attributes, etc.). The home pantry system 252 mayautomatically provide such product recognition criteria to the mobilephone 102 when the associated shopping list is created or downloaded. Abenefit of such automatic product recognition criteria retrieval by thepantry system 252 is that shopper 203 need not explicitly search for ordownload product recognition criteria but can merely visit the store 108and scan the in-store display 104 with the mobile phone 102, a targetproduct 116 on the shopping list automatically being detected andpinpointed by the mobile phone 102.

Example Platform Architecture

FIG. 4 is a network diagram depicting a network system 400, according toone embodiment, having a client-server architecture configured forexchanging data over a network. For example, the network system 400 maybe a publication/publisher system where clients may communicate andexchange data within the network system 400. The data may pertain tovarious functions (e.g., online item purchases) and aspects (e.g.,managing content and user reputation values) associated with the networksystem 400 and its users. Although illustrated herein as a client-serverarchitecture as an example, other embodiments may include other networkarchitectures, such as a peer-to-peer or distributed networkenvironment.

A data exchange platform, in an example form of a network-basedpublisher 402, may provide server-side functionality, via a network 404(e.g., the Internet) to one or more clients. The one or more clients mayinclude users that utilize the network system 400 and more specifically,the network-based publisher 402, to exchange data over the network 404.These transactions may include transmitting, receiving (i.e.,communicating) and processing data to, from, and regarding content andusers of the network system 400. The data may include, but are notlimited to, content and user data such as feedback data; user reputationvalues; user profiles; user attributes; product and service reviews;product, service, manufacture, and vendor recommendations andidentifiers; product and service listings associated with buyers andsellers; auction bids; and transaction data, among other things.

In various embodiments, the data exchanges within the network system 400may be dependent upon user-selected functions available through one ormore client or user interfaces (UIs). The UIs may be associated with aclient machine, such as a client machine 406 using a web client 410. Theweb client 410 may be in communication with the network-based publisher402 via a web server 420. The UIs may also be associated with a clientmachine 408 using a programmatic client 412, such as a clientapplication, or a third party server hosting a third party application.It can be appreciated that in various embodiments the client machine406, 408, or third party application may be associated with a buyer, aseller, a third party electronic commerce platform, a payment serviceprovider, or a shipping service provider, each in communication with thenetwork-based publisher 402 and optionally each other. The buyers andsellers may be any one of individuals, merchants, or service providers,among other things.

A mobile device 432 may also be in communication with the network-basedpublisher 402 via a web server 420. The mobile device 432 may include aportable electronic device providing at least some of thefunctionalities of the client machines 406 and 408. The mobile device432 may include a third party application 416 (or a web client)configured to communicate with application server 422. In oneembodiment, the mobile device 432 includes a GPS module 434 and anoptical lens 436. The GPS module 434 is configured to determine a globallocation of the mobile device 432, for example with reference to globalpositioning satellites.

In some embodiments, the mobile device 432 may include hardware and/orsoftware to provide positioning services at a more granular level thanis the case with typical GPSs. The device 432 may, for example, includean indoor positioning system (IPS) or a real-time locating system (RTLS)to determine the indoor location of the device. Such an IPS may functionto wirelessly establish the position of the device 432 inside a storebuilding. Instead of using satellites, such an IPS may rely on nearbyanchors (e.g., nodes with a known position), which effectively locatetags or provide environmental context for devices to sense. Example IPSsoperate by the use of optical signals, radio signals, acoustic(typically high-frequency) signals, or combinations thereof.

The optical lens 436 enables the mobile device 432 to take pictures andvideos.

Turning specifically to the network-based publisher 402, an applicationprogram interface (API) server 418 and a web server 420 are coupled to,and provide programmatic and web interfaces respectively to, one or moreapplication servers 422. The application servers 422 host one or morepublication application(s) 424 and video processor application(s) 430.The application servers 422 are, in turn, shown to be coupled to one ormore database server(s) 426 that facilitate access to one or moredatabase(s) 428.

In one embodiment, the web server 420 and the API server 418 communicateand receive data pertaining to listings, transactions, and feedback,among other things, via various user input tools. For example, the webserver 420 may send and receive data to and from a toolbar or webpage ona browser application (e.g., web client 410) operating on a clientmachine (e.g., client machine 406). The API server 418 may send andreceive data to and from an application (e.g., programmatic client 412or third party application 416) running on another client machine (e.g.,client machine 408 or third party server 414).

Publication application(s) 424 may provide a number of functions andservices (e.g., listing, payment, store inventory, product querying,product recognition criteria provision, automated product detection,etc.) to users that access the network-based publisher 402. For example,the application(s) 424 may provide a number of services and functions tousers for listing goods and/or services for sale, facilitatingtransactions, and reviewing and providing feedback about transactionsand associated users. Additionally, the application(s) 424 may track andstore data and metadata relating to listings, transactions, and userinteraction with the network-based publisher 402. The application(s) 424may also provide automated product detection application(s) 206 asdescribed, to deliver associated functionalities to a user, and may insome embodiments provide the functionalities described with reference tothe store computer system 218 in FIG. 2.

FIG. 4 also illustrates a third party application that may execute on athird party server 414 and may have programmatic access to thenetwork-based publisher 402 via the programmatic interface provided bythe API server 418. For example, the third party application 416 may useinformation retrieved from the network-based publisher 402 to supportone or more features or functions on a website hosted by the thirdparty.

Example Mobile Device

FIG. 5 is a block diagram illustrating a further example embodiment of amobile device 500 that may be employed to provide the functionalitiesdescribed, e.g., with reference to a mobile phone 102 (see FIGS. 1 and2). According to an example embodiment, the mobile device 500 mayinclude a processor 510 and has an image capture device in the form of acamera 215 that can capture both digital photos and/or digital videocontent. The processor 510 may be any of a variety of different types ofcommercially available processors suitable for mobile devices (forexample, an XScale architecture microprocessor, a Microprocessor withoutInterlocked Pipeline Stages (MIPS) architecture processor, or anothertype of processor). A memory 520, such as a Random Access Memory (RAM),a Flash memory, or other type of memory, is typically accessible to theprocessor 510. The memory 520 may be adapted to store an operatingsystem (OS) 540, as well as application programs 512, such as theabove-discussed in-store product detection application(s) 206, and alocation enabled application that may provide LBSs to a shopper. Ascanning layer 530 may be provided for scanning visual codes based onimage information capture via the camera 215. The processor 510 may becoupled, either directly or via appropriate intermediary hardware, to adisplay 550 and to one or more input/output (I/O) devices 560, such as akeypad, a touch panel sensor, a microphone, and the like. Similarly, insome embodiments, the processor 510 may be coupled to a transceiver 570that interfaces with an antenna 590. The transceiver 570 may beconfigured to both transmit and receive cellular network signals,wireless data signals, or other types of signals via the antenna 590,depending on the nature of the mobile device 500. In this manner, aconnection with the communication network 244 may be established.Further, in some configurations, a GPS receiver 580 may also make use ofthe antenna 590 to receive GPS signals.

Example Methods

FIG. 6 is a schematic flow chart that illustrates an example method 600for automated in-store product detection.

The method 600 may comprise creating a search query, at operation 603,for one or more target products 116; submitting the search query, atoperation 621, to the store computer system 218 or an equivalent onlinewebsite; and receiving product recognition criteria, at operation 635,responsive to the search query.

The product recognition criteria thus retrieved may be stored on themobile phone 102 (see FIG. 2).

Thereafter, when visiting the brick-and-mortar store 108, the shopper203 may capture merchandise display data, at operation 639, of thein-store product display 104 (see FIG. 1). The merchandise display datathus captured may then be used in combination with the previouslyretrieved product recognition criteria for automated in-store productdetection, at operation 655, which may include alerting the shopper 203to the particular location of one or more of the target products 116 bydisplaying a location indicator 128 (FIG. 1) on the phone's screen 120.

Creation of the search query, at operation 603, may be performed eitherat a location remote from the store 108 (e.g., that the shopper's homecomputer system 248), or may be performed while the shopper 203 is inthe store 108. In the latter instance, the creation and submission ofthe search query may be by establishing communication between the mobilephone 102 and the store computer system 218.

The search query may be created by capturing one or more images of eachtarget product 116, at operation 607. This may comprise taking a digitalphotograph of a sample item (e.g., an empty cereal box or previouslypurchased product item, at home), or may be by taking a picture of atwo-dimensional representation of the product (e.g., a magazine orbrochure picture).

Instead, or in addition, one or more target products 116 may be includedin the search query by scanning the associated product bar code, atoperation 611. Again, this may comprise scanning a barcode 132 on asample product item, or scanning a barcode or other visual code separatefrom a product item (e.g., in a magazine, coupon, or the like). Yetfurther, desired target products 116 may be included in the search queryby user-provision of text input, at operation 615.

If a shopping list comprising multiple target products 116 is to becreated, then the search query input is repeated, at operation 619, foreach respective target product 116.

The search query may be submitted (at operation 621) to the storecomputer system 218 or to a merchant or store website, at which aproduct database (e.g., 221) may be accessed to retrieve productrecognition criteria for the target products 116 in the search query. Insome instances, creation of the shopping list may be by online browsingof a website that allows search and selection of desired target products116.

Instead of user-creation of a shopping list, or in addition thereto, oneor more target products 116 may automatically be added to the shoppinglist, e.g., by a home pantry system 252 (see FIG. 2). The pantry system252 may thus monitor pantry stock levels, at operation 625, for multiplerespective products, and may automatically add one or more products tothe shopping list as target products, at operation 631, responsive todetermining, at operation 629, that stock levels have fallen below apredetermined threshold. In some examples, target products 116 may alsobe automatically included in the shopping list at the store 108 (e.g.,by the store computer system 218) based on predefined criteria. Theshopper 203 may, e.g., register to automatically be alerted to productsthat are available at a special deal, for example being priced more thana certain percentage less than usual.

Capturing of the merchandise display data at the store 108 by use of themobile phone 102 (at operation 639), may be by capturing video content,at operation 643; capturing digital photographs, at operation 647; orcapturing identifier signals (e.g., RFID signals) of a target zone ofthe in-store product display 104, at operation 651.

As mentioned before, the automated product detection, at operation 655,may be performed on the mobile phone 102 or, in some embodiments,remotely from the phone 102. This comprises receiving the capturedmerchandise display data, at operation 659, and processing or parsingthe merchandise display data, at operation 663, to detect a particulartarget product 116 indicated by the merchandise display data. Ininstances where the shopper 203 has created a shopping list with aplurality of target products 116, the merchandise display data may beprocessed, at operation 661, for detecting any or each of the respectivetarget products 116 in the shopping list. Two or more target products116 may thus be recognized and indicated at any particular moment.

A graphical view of the portion of the in-store product display 104captured by the merchandise display data (e.g., an augmented realityview 124 such as that shown in FIG. 1) may be displayed on the screen120 of the mobile phone 102, at operation 667.

Displaying a graphical view may include, at operation 671, providingrespective location indicators 128 (see FIG. 1) for each target product116 that is identified and that is present in the target zone currentlydisplayed on the screen 120.

It is a benefit of the above-described method that it promotes readyuser-discovery of a desired product in a store environment, with minimaluser effort. Automatic shopping list creation may further facilitatepantry stock level management and, to the benefit of merchants, promoteregular stock replenishment.

Components, and Logic of Example Embodiments

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules, with code embodied on a non-transitorymachine-readable medium (i.e., such as any conventional storage device,such as volatile or non-volatile memory, disk drives or solid statestorage devices (SSDs), etc.), or hardware-implemented modules. Ahardware-implemented module is a tangible unit capable of performingcertain operations and may be configured or arranged in a certainmanner. In example embodiments, one or more computer systems (e.g., astandalone, client, or server computer system) or one or more processorsmay be configured by software (e.g., an application or applicationportion) as a hardware-implemented module that operates to performcertain operations as described herein.

In various embodiments, a hardware-implemented module may be implementedmechanically or electronically. For example, a hardware-implementedmodule may comprise dedicated circuitry or logic that is permanentlyconfigured (e.g., as a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an application-specific integratedcircuit (ASIC)) to perform certain operations. A hardware-implementedmodule may also comprise programmable logic or circuitry (e.g., asencompassed within a general-purpose processor or other programmableprocessor) that is temporarily configured by software to perform certainoperations. It will be appreciated that the decision to implement ahardware-implemented module mechanically, in dedicated and permanentlyconfigured circuitry or in temporarily configured circuitry (e.g.,configured by software), may be driven by cost and time considerations.

Accordingly, the term “hardware-implemented module” should be understoodto encompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarily ortransitorily configured (e.g., programmed) to operate in a certainmanner and/or to perform certain operations described herein.Considering embodiments in which hardware-implemented modules aretemporarily configured (e.g., programmed), each of thehardware-implemented modules need not be configured or instantiated atany one instance in time. For example, where the hardware-implementedmodules comprise a general-purpose processor configured using software,the general-purpose processor may be configured as respective differenthardware-implemented modules at different times. Software mayaccordingly configure a processor, for example, to constitute aparticular hardware-implemented module at one instance of time and toconstitute a different hardware-implemented module at a differentinstance of time.

Hardware-implemented modules can provide information to, and receiveinformation from, other hardware-implemented modules. Accordingly, thedescribed hardware-implemented modules may be regarded as beingcommunicatively coupled. Where multiple of such hardware-implementedmodules exist contemporaneously, communications may be achieved throughsignal transmission (e.g., over appropriate circuits and buses) thatconnect the hardware-implemented modules. In embodiments in whichmultiple hardware-implemented modules are configured or instantiated atdifferent times, communications between such hardware-implementedmodules may be achieved, for example, through the storage and retrievalof information in memory structures to which the multiplehardware-implemented modules have access. For example, onehardware-implemented module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware-implemented module may then,at a later time, access the memory device to retrieve and process thestored output. Hardware-implemented modules may also initiatecommunications with input or output devices, and can operate on aresource (e.g., a collection of information).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), with these operations being accessiblevia a network (e.g., the Internet) and via one or more appropriateinterfaces (e.g., Application Program Interfaces (APIs).)

FIG. 7 shows a diagrammatic representation of a machine in the exampleform of a computer system 700 within which there may be executed a setof instructions 724 for causing the machine to perform any one or moreof the methodologies discussed herein. For example, the system 100(FIG. 1) or any one or more of its components (FIGS. 1 and 2) may beprovided by the system 700.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a server computer, a client computer, a personal computer(PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant(PDA), a cellular telephone, a web appliance, a network router, switchor bridge, or any machine capable of executing a set of instructions(sequential or otherwise) that specify actions to be taken by thatmachine. Further, while only a single machine is illustrated, the term“machine” shall also be taken to include any collection of machines thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein.

The example computer system 700 includes a processor 702 (e.g., acentral processing unit (CPU) a graphics processing unit (GPU) or both),a main memory 704 and a static memory 706, which communicate with eachother via a bus 708. The computer system 700 may further include a videodisplay unit 710 (e.g., a liquid crystal display (LCD) or a cathode raytube (CRT)). The computer system 700 also includes an alpha-numericinput device 712 (e.g., a keyboard), a cursor control device 714 (e.g.,a mouse), a disk drive unit 716, an audio/video signal input/outputdevice 718 (e.g., a microphone/speaker) and a network interface device720.

The disk drive unit 716 includes a machine-readable storage medium 722on which is stored one or more sets of instructions 724 (e.g., software)embodying any one or more of the methodologies or functions describedherein. The instructions 724 may also reside, completely or at leastpartially, within the main memory 704 and/or within the processor 702during execution thereof by the computer system 700, the main memory 704and the processor 702 also constituting non-transitory machine-readablemedia.

The instructions 724 may further be transmitted or received over anetwork 726 via the network interface device 720.

While the machine-readable storage medium 722 is shown in an exampleembodiment to be a single medium, the term “machine-readable medium”should be taken to include a single medium or multiple media (e.g., acentralized or distributed database and/or associated caches andservers) that store the one or more sets of instructions 724. The term“machine-readable medium” shall also be taken to include any medium thatis capable of storing a set of instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of this disclosure. The term “machine-readable medium”shall accordingly be taken to include, but not be limited to,solid-state memory devices of all types, as well as optical and magneticmedia.

Thus, systems and methods for in-store product detection are disclosed.Although these methods and systems have been described with reference tospecific example embodiments, it will be evident that variousmodifications and changes may be made to these embodiments withoutdeparting from the broader spirit and scope thereof. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, the disclosed subject matter lies in less than all features ofa single disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment.

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesor objects that operate to perform one or more operations or functions.The modules and objects referred to herein may, in some exampleembodiments, comprise processor-implemented modules and/or objects.

What is claimed is:
 1. A system comprising: a non-transitory memory; andone or more hardware processors coupled to the non-transitory memory andconfigured to read instructions from the non-transitory memory to causethe system to perform operations comprising: retrieving a list of targetproducts, wherein the list includes product recognition criteria foreach of the target products; receiving, from a store computer system,locations of one or more of the target products; capturing, using acamera on a mobile computing device, digital images of products ondisplay at a physical store; extracting, from the captured digitalimages of the products on display at the physical store when the mobilecomputing device is determined to have entered a location correspondingto the one or more of the target products, attributes that correspond toproduct recognition criteria for the one or more of the target products;and displaying, on a display of the mobile computing device, a modifiedreal time representation of the products on display at the physicalstore, wherein the modified real time representation of the products ondisplay includes augmented reality location indicators overlayed on asubset of the one or more of the target products located within a fieldof view of the camera.
 2. The system of claim 1, wherein the operationsfor displaying the modified real time representation of the products ondisplay at the physical store comprise: determining an on-screenposition of the subset of the one or more of the target products withinthe field of view of the camera; and modifying the real timerepresentation of the products on display by overlaying the locationindicators over the subset of the one or more of the target products. 3.The system of claim 1, wherein the product recognition criteria for eachof the target products include at least one of barcodes, definitions ofone or more visual characteristics or visual indicators, or opticalproduct recognition attributes.
 4. The system of claim 3, wherein theone or more visual characteristics or visual indicators correspond totrade dress elements uniquely associated with the target products. 5.The system of claim 1, wherein the operations further comprisedisplaying one or more direction indicators indicating positions ofadditional target products in the physical store, the directionindicators being displayed relative to a physical geometry of the realtime representation.
 6. The system of claim 5, wherein the directionindicators are generated based on mapping information of the physicalstore provided by the store computer system.
 7. The system of claim 1,wherein the list of target products is created on the user account by ahome pantry system.
 8. The system of claim 7, wherein the targetproducts are automatically added to the list based on a monitoring ofpantry stock levels, the target products being added when correspondingstock levels fall below a predetermined threshold.
 9. A methodcomprising: creating a list of target products created, wherein the listincludes product recognition criteria for each of the target products;receiving, from a store computer system, prompts for navigating tolocations of one or more of the target products; capturing, using acamera on a mobile computing device, digital images of products ondisplay at a physical store; extracting, from the captured digitalimages of the products on display at the physical store when the mobilecomputing device is determined to have entered one of the locationscorresponding to the one or more of the target products, attributes thatcorrespond to product recognition criteria for the one or more of thetarget products; and displaying, on a display of the mobile computingdevice, a modified real time representation of the products on displayat the physical store, wherein the modified real time representation ofthe products on display includes augmented reality location indicatorsoverlayed on a subset of the one or more of the target products locatedwithin a field of view of the camera.
 10. The method of claim 9, whereinthe displaying the modified real time representation of the products ondisplay at the physical store comprises: determining an on-screenposition of the subset of the one or more of the target products withinthe field of view of the camera; and modifying the real timerepresentation of the products on display by overlaying the locationindicators over the subset of the one or more of the target products.11. The method of claim 9, wherein the product recognition criteria foreach of the target products include at least one of barcodes,definitions of one or more visual characteristics or visual indicators,or optical product recognition attributes.
 12. The method of claim 11,wherein the one or more visual characteristics or visual indicatorscorrespond to trade dress elements uniquely associated with the targetproducts.
 13. The method of claim 9, wherein the prompts for navigatingto the locations of the one or more of the target products include adirection indicator for indicating a positions of the one or more of thetarget products in the physical store, the direction indicators beingdisplayed relative to a physical geometry of the real timerepresentation.
 14. The method of claim 13, wherein the directionindicators are generated based on mapping information of the physicalstore provided by the store computer system.
 15. A non-transitorycomputer-readable medium containing instructions which, in response tobeing executed by one or more processors, cause a portable electronicdevice to perform operations comprising: retrieving a list of targetproducts, wherein the list includes product recognition criteria foreach of the target products; receiving, from a store computer system,locations of one or more of the target products; capturing, using acamera on a mobile computing device, digital images of products ondisplay at a physical store; extracting, from the captured digitalimages of the products on display at the physical store when the mobilecomputing device is determined to have entered a location correspondingto the one or more of the target products, attributes that correspond toproduct recognition criteria for the one or more of the target products;and displaying, on a display of the mobile computing device, a modifiedreal time representation of the products on display at the physicalstore, wherein the modified real time representation of the products ondisplay includes augmented reality location indicators overlayed on asubset of the one or more of the target products located within a fieldof view of the camera.
 16. The non-transitory computer-readable mediumof claim 15, wherein the operations for displaying the modified realtime representation of the products on display at the physical storecomprise: determining an on-screen position of the subset of the one ormore of the target products within the field of view of the camera; andmodifying the real time representation of the products on display byoverlaying the location indicators over the subset of the one or more ofthe target products.
 17. The non-transitory computer-readable medium ofclaim 15, wherein the product recognition criteria for each of thetarget products include at least one of barcodes, definitions of one ormore visual characteristics or visual indicators, or optical productrecognition attributes.
 18. The non-transitory computer-readable mediumof claim 17, wherein the one or more visual characteristics or visualindicators correspond to trade dress elements uniquely associated withthe target products.
 19. The non-transitory computer-readable medium ofclaim 15, wherein the list of target products is created on the useraccount by a home pantry system.
 20. The non-transitorycomputer-readable medium of claim 19, wherein the target products areautomatically added to the list based on a monitoring of pantry stocklevels, the target products being added when corresponding stock levelsfall below a predetermined threshold.